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Abstract 

Eleven Canadian research granting agencies were surveyed concerning the 
weights that they apply to various criteria when reviewing applications for 
graduate student awards. Nine agencies responded, seven providing useful data. 
The undergraduate academic grades of applicants for research training awards 
were found to carry high weight in selection decisions, especially in 
competitions that were either large or were open to students at the master’s 
level. However, the literature on early career indicators suggests that the 
undergraduate academic achievement of graduate students is not an effective 
predictor of their future research productivity. Strategies through which 
agencies might improve the use of more predictive criteria, such as applicant 
characteristics and quality of the graduate training environment, are discussed. 


Resume 

Onze organisations suhventionnaires canadiennes ont fait I’objet d’une enquete 
concernant le poids qu’elles attribuent a divers criteres dans leur evaluation de 
demandes de bourses. Neuf questionnaires ont ete retournes; sept d’entre eux 
ont pu etre utilises. Les notes de premier cycle obtenues par les candidats de 
bourses de recherche revetaient une grande importance dans les decisions, 
surtout dans les concours ouverts aux etudiants de deuxieme cycle. Cependant, 
la litterature sur les indicateurs precoces d’une carriere scientifique suggere 
que les notes de premier cycle obtenues par les etudiants gradues ne sont pas 
un bon indicateur de leur productivity future en recherche. Les strategies 
susceptibles d’augmenter T utilisation des criteres plus fiables, comme les 
caracteristiques des candidats et la qualite de leur environnement de recherche, 
sont discutees dans cet article. 
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Introduction 

Research scholarships provide graduate students with both financial support and 
societal recognition of the importance of their work and are vital to the 
development of a national cadre of highly qualified research personnel. For 
47% of Ph.D. graduates holding appointments in health science faculties, 
research scholarships were the principal source of support during graduate study 
(MRC, 1988a). The importance of graduate research scholarships in the 
development of a nation-wide research capability justifies careful examination 
of the criteria by which they are awarded. This study presents information on 
the weightings that granting agencies assign to various selection criteria and 
discusses the appropriateness of those weightings given the theoretical 
predictive strength of each criterion. 

Criteria often used in the selection of graduate student award winners include 
the undergraduate academic grades of applicants, assessments by sponsors, the 
quality of the proposed graduate research project and training environment, and 
the applicants’ traits and research experience. The validity of some of these 
criteria is questionable. The correlation between the undergraduate academic 
grades of graduate students and their eventual research productivity is very low 
(Clark and Centra, 1982; O’Brecht, Pihl and Bois, 1989). While students with 
high grades may be more likely to obtain admission to prestigious universities, 
and graduates of prestigious universities may be more likely to end up in careers 
where research is a major activity, the link between undergraduate academic 
performance and career productivity is tenuous at best. Since the range of 
undergraduate grades in a graduate student population is very narrow, 
essentially restricted to A and B records, it is not surprising that grades are a 
poor discriminator of research potential in an applicant pool comprised of 
graduate students. Sponsors’ assessments of applicants for graduate awards also 
appear to be poor predictors of the level of post-training research activity. This 
is possibly because sponsors’ assessments are highly correlated with applicant’s 
undergraduate academic performance (Davis, 1965; O'Brecht, et al., 1989). The 
appropriateness of assessing the quality of the graduate research proposal when 
reviewing applications for research training awards has received little attention 
in studies of research career indicators. However, a positive correlation between 
number of publications resulting from graduate work and post-training research 
productivity suggests that the scientific import of the graduate research project 
is a relevant criteria. Further, many studies have found a positive correlation 
between the quality of the graduate training environment and subsequent 
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research activity (Folger, Astin and Bayer, 1970; Clemente, 1973; and Zumeta, 
1983). In the health sciences, the research orientation of the immediate 
environment for graduate training appears to be especially critical (O’Brecht, et 
al., 1989). That is, students training under active researchers in an atmosphere 
where research is highly valued and often talked about, where travel to 
scientific meetings is expected and frequent, where guest lecturers and 
postdoctoral trainees abound, are more likely to become productive researchers 
than students who train in less stimulating environments. A number of 
personality traits also appear to be correlated with research productivity. 
Independence, critical ability and determination all seem to be good predictors 
(O’Brecht, et al., 1989). Pre-university interest in science also showed a 
positive correlation with research career outcome in a population of biological 
scientists (Segal, Busse and Mansfield, 1980). Research experience during 
undergraduate study was found to bear a significant correlation with 
post-training research activity in a sample of health professionals (O’Brecht 
and Bois, 1988). However, in a population of graduate students in the health 
sciences (a population which includes persons with science, social science, 
engineering or arts undergraduate degrees) research experience during 
undergraduate study does not appear to be an important predictor, possibly 
because undergraduate research awards tend to be distributed on the basis of 
academic course results (O’Brecht, et al., 1989). 

In summary, the literature on early indicators of research careers would 
suggest that agencies wishing to identify graduate students who are likely to 
become productive researchers should focus their attention on the applicants’ 
traits, interests and training environment. The present study provides 
information on the current selection practices of a variety of granting agencies 
in both the public and private sectors. 

Method 

Selection of Sample 

The population of interest included agencies with a specific mandate to support 
research (e.g., research granting councils). Programs intended to support 
graduate study under an educational mandate (e.g., provincial graduate 
scholarships programs) were not included. 

The Reference List of Health Science Research in Canada and other public 
directories were examined to identify granting agencies that operate graduate 
student award programs. The sample included three private non-profit agencies 
(the Heart and Stroke Foundation of Canada, the National Cancer Institute of 
Canada and the Muscular Dystrophy Association of Canada), four provincial 
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agencies (the Alberta Heritage Foundation for Medical Research, the Fonds de 
recherche en sante) du Quebec, the Ontario Mental Health Foundation and the 
Ontario Ministry of Health), and four federal agencies (the National Health 
Research and Development Program, the Medical Research Council, the 
Natural Sciences and Engineering Research Council, and the Social Sciences 
and Humanities Research Council). 

Collection of Data 

Questionnaires, in both English and French, were developed to solicit 
information from agency managers on the number of applications received, 
awards granted, procedures for reviewing applications and weightings assigned 
to various selection criteria. Separate data were collected for programs open to 
master’s students and those restricted to doctoral students. Questionnaires were 
mailed in August of 1989 and agencies that had not responded by October were 
telephoned and their participation again requested. Agencies were asked to 
complete the questionnaire and to provide copies of their application material as 
well as any documentation on the process by which they select graduate student 
award winners. 

Definition of Data 

The six categories of selection criteria presented in the questionnaire were as 
follows: 1) quality of the graduate training environment , a category that 
included such variables as the productivity of the applicant’s graduate 
supervisor, the reputation and research funding of the training laboratory and 
opportunities for collaboration with other trainees and investigators; 2) the 
quality of the project proposed bv the applicant and the level of his/her input 
into development of the project; 3) traits of the applicant , including 
characteristics such as the candidate’s independence, determination, interest in 
research, and so forth; 4) academic achievement of the applicant , primarily 
reflecting grades achieved in university courses; 5) research experience of the 
applicant , especially the extent to which the applicant's undergraduate (or early 
graduate) research experience is considered in the selection process; and, 6) 
other , a free response category to ensure that agencies would report all selection 
criteria. 

Results 

Nine of the research granting agencies agreed to participate in the study and 
useful data were received from seven. (One of the provincial agencies had 
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recently closed its program to new applicants and a federal government agency 
was not able to estimate criteria weights because of the nature of its review 
process.) A statistical profile of the seven agencies appears in Table 1 . 

Table 2 shows the data provided by seven granting agencies in response to 
the question on weighting of each criterion. (Two agencies, A and D, preferred 
to group several criteria together when providing estimates.) Results indicate 
that assessment of the applicant’s personal file (traits, grades obtained and 
experience) carries approximately 70% of the weight in a typical selection, 
while assessment of the proposed graduate research training (training 
environment, research project and other) receives a weight of about 30%. 
However, there is wide variation among agencies in the relative importance 
attributed to assessment of the individual and the project. At one extreme, a 
small agency assigns only a 40% weight to assessment of the applicant; at the 
other extreme, a large agency assigns 100% weight. 

Academic marks tend to be the most important criterion. For awards 
programs that are open to master’s students, course results account for about 
68% of the rating. Academic grades remain an important criterion in 
competitions restricted to Ph.D. students, but carry noticeably less weight (47%) 
than in competitions open to master’s students. In competitions restricted to 
Ph.D. students, more weight is placed on the personal traits of candidates and 
the quality of research proposals. These differences are illustrated in Figure 1 . 

As an examination of the raw data suggested that larger agencies attribute 
more weight to academic grades than do smaller agencies, correlations between 
size of agency (number of applicants) and criteria weights were computed. The 
results appear in Table 3 . Data confirm a positive correlation between agency 
size and the amount of emphasis placed on academic course results. In 
comparison with smaller agencies, larger agencies will tend to assign more 
weight to the applicant’s grades and less weight to the research project. 

As the intent of the data collection exercise was primarily to expose the 
range of weights applied to criteria by the various agencies, and since the 
number of agencies in the study was not large, further statistical analysis of the 
data was deemed inappropriate. 

Discussion 

Selection Theory and Agency Practice 

Research granting agencies may pursue a variety of objectives through their 
graduate awards programs. The awards may be intended to recognize academic 
excellence, to produce master’s or doctoral students or to support the training of 
future researchers. However, regardless of the stated program objectives, or lack 



Table 1 

Profile of Agencies that Provided Data on Weightings of Selection Criteria 


Characteristics Agency 



A 

B 

c 

D 

E 

F 

G 

Sector 1 

P-n-p 

P-n-p 

Prov 

Prov 

Prov 

Fed 

Fed 

Programs^ 

M/PhD 

PhD 

M;PhD 

M/PhD 

M/PhD 

M;PhD 

M/PhD 

Applicants^ 

122 

44 

302; 189 

38 

188 

2545; 1174 

877 

Award rate^ 

19% 

11% 

9% 

21% 

29% 

49% 

20% 

External Prescreening^ 

none 

unknown 

unknown 

minor 

minor 

major 

none 

Internal Prescreening^ 

no 

yes 

yes 

no 

no 

no 

no 

Interviews 

no 

yes 

no 

no 

no 

no 

no 


1: P-n-p, Private-non-profit; Prov, Provincial government; Fed, Federal government. 

2: M/Ph.D., program open to master’s or doctoral students; M, open only to master’s degree students; PhD, 

open only to doctoral students. 

3: Total number of applicants for each program. 

4: Overall approval rate for the agencies’ graduate award programs. 

5: External pre-screening of applications: minor, agencies report that university departments may screen 
applications before they are submitted for consideration; major, universities review applications and 
submit a ranked list. 

6: Internal pre-screening of applications, i.e., a two-stage selection process within the granting agency, an 
initial screening followed by a final selection. 
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Table 2 

Weighting of Various Criteria in the Selection of Graduate Research Award 
Recipients 


Criterion * Agency and Percentage Weights for Criteria 

(by program type) 


Programs Open to 
Masters Students 

A 

B 

C 

D 

E 

F 

G 

Average 

Weight 

(1) Environment 

40 


16.7 

_ 

20 

0 

5 

10.4 

(2) Project 

20 


16.7 

- 

15 

0 

0 

7.9 

(3) Traits 

_2 


16.7 

- 

5 

10 

5 

9.2 

(4) Academic 

- 


50 

- 

50 

80 

90 

67.5 

(5) Experience 

- 


0 

- 

10 

0 

0 

2.5 

(6) Other 

0 


0 

- 

0 

10 

0 

2.5 

(1) and (2) 

60 


33 

40 

35 

0 

5 

28.8 

(3), (4), (5) and (6) 3 

40 


67 

60 

65 

100 

95 

71.2 


Programs Open 

Onlv to PhD Students 





(1) Environment 

10 

25 

0 

11.7 

(2) Project 

20 

25 

0 

15.0 

(3) Traits 

25 

25 

10 

20.0 

(4) Academic 

35 

25 

80 

46.7 

(5) Experience 

5 

0 

0 

1.7 

(6) Other 

5 

0 

10 

5.0 

(1) and (2) 

30 

50 

0 

26.7 

(3), (4), (5) and (6) 

70 

50 

100 

73.3 


1: Environment, graduate training environment; project, graduate research project; 
traits, personal characteristics of applicant; academic, academic grades; experience, 
previous research experience. 


2: The agency did not estimate weights for the specific criterion but did estimate the 
weight for a grouping of criteria that included the criterion. 

3: “Other” criteria related to assessment of applicant qualities. In one case “other” was 
a small score related to the impact of an interview with the candidate; in another 
case, “other” was the assessment and ranking provided by the chair of the 
applicant’s department of major study. 
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Table 3 

Correlation Coefficients Between Selection Criteria and Size of Competition 


Criterion 

Correlation 1 with Size of Competition by Program Type 


Open to 

PhD 


Master’s 

Students 


Students 

Only 


(max n=6) 

(n=3) 

1) Training Environment 

-.77 

-.73 

2) Research Project 

-.81 

-.99 

3) Applicant Traits 

-.02 

-.99 

4) Academic Grades 

.65 

.95 

5) Research Experience 

-.49 

-.60 

6) Other 

.95 

.80 

Proposal (1 and 2) 

-.81 

-.86 

Applicant (3, 4, 5 and 6) 

.80 

.86 


1: R values calculated using the correlations procedure of SPSS-PC. 


Figure 1 

Selection Crit eria an d We igh ts 

Weight in Selection Process (%) 



GRADES ENVIRONMENT PROJECT TRAITS M1SC.** 

* Master’s or master’s/doctoral 
** Research experience and “other” 
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thereof, it is reasonable to assume that research granting agencies will be 
interested in supporting students who are likely to become active researchers 
following their graduate and postdoctoral training. For example, in a survey of 
panel chairmen and program administrators concerning the graduate Studentship 
program of the Medical Research Council, the objective “to support the training 
of future researchers”, although not formally stated in the program guidelines, 
received the highest mean rating of importance of five statements of program 
objectives (Medical Research Council, 1986). 

Findings from the research literature would suggest that agencies which are 
interested in identifying and supporting future researchers should not restrict 
their review of applicants to a consideration of undergraduate academic records 
but should also carefully consider the personal traits of applicants and their 
graduate training environment. In practice, research funding agencies weight 
undergraduate academic grades more heavily than other criteria. For all but one 
of the agencies in this study, academic grades carried the largest weight in the 
selection process. 

At least two factors influence the weight that awards committees attach to 
academic grades: the level of study of the applicants and the number of 
applications to be reviewed. First, if the awards competition is open to master’s 
level students, reliance on academic grades will tend to be high. Candidates for 
master’s level awards, often applying while still in their undergraduate program 
are, arguably, not in a good position to develop a sophisticated research 
proposal. In fact, some may not have finally decided upon the location for their 
graduate work. In addition, undergraduate students have not had a full 
opportunity to demonstrate traits that relate to research productivity 
(independence, critical ability and determination). In effect, inadequate data on 
the graduate research environment and applicant traits are a probable cause of 
heavy reliance on grades in the assessment of applications from students at the 
master’s level. For awards competitions that are restricted to doctoral degree 
students, assessment of the applicants’ research proposals and research-related 
traits becomes more feasible. A second influence on the weighting of academic 
grades in the award selection process is the number of applicants that must be 
reviewed. The larger the number of applications, the greater the reliance on 
academic marks as a selection criterion. Decisions based primarily on academic 
grades, as compared with decisions that take into account the individual 
qualities and training situations of applicants, can be made quickly and 
efficiently and yet will appear to be fair and rational. It is therefore not 
surprising that agencies which must review a large number of applications will 
tend to assign high weight to the academic achievements of candidates. 
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Importance of Optimal Weighting of Selection Criteria 

The number of Canadian students now in or entering the science and technology 
research training pipeline appears to be insufficient to meet expected future 
demand (Natural Sciences and Engineering Research Council, 1989; Medical 
Research Council, 1988b). Data on the projected balance between the supply 
and demand of scientists in the United States indicate a future undersupply that 
should be quite evident by 1995 and of major proportion by the turn of the 
century (Atkinson, 1990). A shortage of scientists in the U.S. would probably 
worsen the problem in Canada. If Canadian and U.S. predictions of demand for 
research personnel are accurate, the identification of the most promising future 
scientists within pools of applicants for graduate student awards becomes an 
increasingly important exercise. 

The effectiveness of research training programs is directly dependent on the 
quality of awardee selection processes. A less than optimal selection process 
implies that some resources will be awarded to trainees who are unlikely to 
attain program objectives, while better prospects, or applicants to other 
programs, must be turned down for lack of funds. Agencies that proclaim the 
pursuit of excellence as their mission must regularly examine the relevance of 
the selection criteria that underlay their concept of excellence. The literature 
suggests that excellence in one area (e.g., academic course performance) does 
not necessarily correlate with excellence in another (e.g., post-training research 
activity). 

Improving the Selection Process 

The research trainee selection process may be improved through a variety of 
measures: agencies may assign more weight to criteria that appear to predict 
desired program outcome; application forms may be revised to collect better 
information on predictor variables; and, the structure of programs or selection 
processes may be changed to facilitate better use of effective predictors. 

Agencies should critically examine the criteria used in their selection of 
graduate student award winners. There is a tendency to assume that historical 
use of predictors validates their effectiveness. Once an agency recognizes that 
criteria may be added, reweighted or dropped, selection processes may be 
rapidly improved. For example, in the mid 1980s, Agency C completely revised 
its graduate student selection criteria, using an intuitive approach rather than 
studying the research literature on early career indicators, but nevertheless 
developing criteria weights that closely approximate theoretical optima. From 
an almost total reliance on academic grades, the agency moved to a selection 
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process in which that criterion accounts for only half of the score for 
applications from master’s students and only one quarter of the score for 
applications from Ph.D. students. 

A better information base on relevant criteria should also improve the quality 
of selection committee decisions. To enable an assessment of the graduate 
training environment for Ph.D. degree students, application forms should collect 
data on the graduate research supervisor: productivity; funding; and, trainees. 
To obtain complete data on the student’s research proposal, agencies should 
request clarification of authorship (applicant or research supervisor), indicate 
the points to be covered in the project description, and provide enough space for 
a full description of the proposed work. 

Assessment of applicant traits is particularly challenging because sponsors’ 
ratings of the personal characteristics of undergraduate students will tend to be 
influenced by their perception of the student’s academic performance. Agencies 
should consider various techniques to enhance the quality of data on applicant 
characteristics. Sponsors’ assessment forms should be carefully structured. The 
forms should ask for data only on key research-related traits: critical judgment, 
independence, determination, and interest in research. Presentation of a long list 
of traits (the typical assessment form lists seven) may encourage a superficial 
response. For each trait, there should be a separate rating scale and a space for 
sponsors to state the reason for their rating. Also, agencies should consider 
obtaining reasonably extensive (e.g., one page) statements from candidates 
about their interests and career goals. 

The present study suggests that when agencies use one program to support 
both master’s and doctoral students, the selection process tends to favour 
undergraduate grades, a criterion which is even less appropriate for doctoral 
than for master’s students. Agencies should operate separate competitions for 
students with no graduate experience and those that have already completed 
some graduate work. 

For large agencies, the logistics of reviewing hundreds or even thousands of 
applications may prohibit a detailed examination of each case. These agencies 
should adopt a pre-screening procedure, eliminating on a first round of review 
those applications which are clearly not competitive. The remaining 
applications should then be assessed using all criteria found to be predictive of 
the desired program outcome. 
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Conclusion 

There appears to be considerable opportunity for agencies to enhance the 
effectiveness of their granting programs by improving selection processes. 
Agencies may revise criteria, adjust importance weights, rework application 
forms or stream applicants to sharpen the judgment process and thus increase 
the probability of supporting individuals who are most likely to achieve 
program objectives. 
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